Hello from the 90s

My name is Howard Butler, and I don’t know very much JavaScript, I think I’m here because I’m good at tweeting pithy in-jokes for geospatial programmers who came up in the late 90s. We all know how well choosing people because of their tweeting abilities goes.

I hope to provide more than tweet sized nuggets about platforms for geospatial software,

overlay my software development history with recent GIS software history

describe how platforms are mixing together faster than they were in the 90s and 2000s

describe how good open source software systems build with rather than instead of

and talk about how this all relates to our now favorite computing platform

– or at least the one for this conference – JavaScript

Talking about the 90s also gives opportunity to cargo shorts in front of an audience.

The foundation of one software civilization is the bones of another.

Like many, I got to geospatial software development in an unplanned and happenstance way.

While I had the 80s nerd starter kit of an Apple IIgs with BASIC, the reality that I would become someone who makes their living giving away software is still astonishing to me.

I grew up on a corn and soybean farm in Southern Minnesota, and I spent many summer days and fall nights in the fields, helping to plant, maintain, or harvest crops.

Throughout high school, I was given more responsibility and information about how to successfully grow a crop. Scratching out a living in the dirt was not just a metaphor, and I went to university looking to become better at it.

I bounced around a few majors in college, including some computer engineering, before I settled on agriculture technology development. I even took a course series at Iowa State University conveniently nicknamed

Dirt,

Super Dirt,

and Super Duper Dirt.

Getting closer to graduation, I started looking into technologies that could really make a difference for crop farming in the upper Midwest,

I came across a software program that would make that all crystal clear for me

ArcView

ArcView gave me the ability to overlay whatever I wanted.

It’s like an automatic spurious relationship finder, especially in the hands of a newbie, and I was fantastic at using it to create horrific cartography and compute indescribable statistics of mostly just population maps of corn.

Eventually, button and menu functionality weren’t enough for me or my major professor, and I needed script to graduate. Avenue my first real platform

Avenue was a SmallTalk-related language that powered the GUI and data processing objects in ArcView.

I had the basics of software development down –

loops,

conditionals,

and swearing

before I started with ArcView,

but in the course of three or four years, I became a max-level Genji with Avenue.

I could DPS my way through any analysis problem.

I could sling vectors and rasters around with impunity, but despite my skills, I was pretty easy to one-shot.

Avenue was a closed system with a weak software ecosystem.

There was some user-to-user script sharing.

It allowed scripts to call out to shared DLL libraries.

DLLs brought entire software fields to ArcView.

At the ESRI users conference, I saw powerline utilities mapping electrical grid energy state in real time button clicks. God, I know this sounds lame now, but in the time of cargo pants, it really wasn’t.

ESRI became rootbound by the success of ArcView.

The ecosystem integrated poorly with COM Windows connectivity APIs.

If you think my cargo shorts are ugly, you should write some COM software

ESRI pulled a Netscape and rewrote themselves from the ground up to be Windows First and Windows Everywhere.

Along the way they jettisoned my GIS software technician security blanket – Avenue.

During the transition, an ESRI Canada programmer named Bruce Dodson

– he’s still there –

posted a DLL that hooked Avenue to the Python programming language.

Python 2.0 had just come out, and geospatial software in the Python space was dominated by GDAL and PROJ Python APIs

yes I know those Python APIs still look like they did 18 years ago. eh-hem

Python access gave me a transition path that allowed me to build upon my past work while giving me a massive productivity boost in a batteries-included language that also had an open source ethos.

I jumped into Python and wrote an ArcView extension called avTerra that downloaded TMS map tiles into ArcView map views –

except instead of TMS it was called Microsoft TerraServer and there was only one server on the whole internet,

and instead of any sane URLs and JSON, it used this cargo shorts kinda thing called SOAP with XML to do its communication.

Python gave me super powers, and I was able to use them to fill map views with ten-year-old 3 meter black and white imagery.

I still take solace in knowing how many grad students I freed from the drudgery of digitizing from paper maps by allowing them to do heads’ up digitizing in ArcView.

Oh, right, the JavaScript.

Actually, let’s go to Java for a minute first.

The cargo shorts developer crowd was quite excited about the grand promise of Java in the 90s

– write once, run everywhere –

but the devils’ bargain to be struck was rewriting your software to live on the Java Virtual Machine.

Sun being the lucky-but-incompetent Sun held tight to the JVM thinking it was their Twins which they could charge tribute to cross.

Java succeeded poorly for many, especially in the 90s.

Early on, the speed disparity between native code and Java was large.

GUI programming in Java was the normal amount of programming hell plus dealing with Java’s Swing API.

The dominant programming problem of the time was native windows applications, and 90s Java was poorly suited for it.

Java didn’t exactly fail, it’s everywhere even today.

The market transition to Java didn’t go as intended. It didn’t replacing everything and run everywhere.

The bargain had too high a cost – porting working code to new environments creates new bugs in addition to carefully porting over your old ones.

Jumping to the new environment meant tossing out once battle tested designs.

Java poorly allowed systems to build upon their past, and the entire global software library had to be rebuilt from scratch

Decades long investment still hasn’t produced a feature-parity pure Java GDAL or Proj.4 library.

Instead of one-for-one replacing of existing systems, the Java attracted new development that wanted the advantage of the JVM, and older systems stayed where they were – ultimately tied to some C or Fortran compiler to native platforms.

The JavaScript …

Just a sec, let’s talk some more about platforms. In ten years, in 2027, what platforms will still be alive?

Windows should still be here.

Microsoft proved that getting their heads bashed in on mobile can do wonders for their ability to think about software development platforms.

They now make tools that play well with others.

They build on and with things they didn’t actually make or control.

They actually care that you’re using their tools.

The Microsoft in the time of cargo shorts only sweatily claimed they did all this.

Android will still be there.

Android is our friend Java.

It’s even GUI programming in Java too.

Android is a platform with literally a billion users.

iOS of course won’t be dead by then.

iOS has the snobbiest installed base.

Its jail-like pastiche, emphasis on security, and stockhom syndrome users won’t be giving it up.

OSX will be dead by then. OSX is the Lannisters of platforms.

It needs to pay off its technical debt at the Iron Bank of Bravos

It looks pretty but kills everything it touches.

Linux of course.

As a containerized platform, or a platform of containers.

Either way, Linux is the 78 percent nitrogen in the air of platforms – it is so assumed we don’t bother to think about it.

Finally, the one we care about, JavaScript. Every browser has a JavaScript engine, and just about every computer has a browser. JavaScript has billions of users.

Server-side JavaScript is of course a thing too.

This platform is as strong as ever as evidenced by this conference and its evolution from talking about novelty browser mapping capabilities to full scale replacements for the entire spectrum of GIS.

I hate JavaScript –

Ok, maybe that’s a rather cliche thing to say.

Maybe you all hate JavaScript too.

I don’t hate JavaScript because the language is awful and arbitrary.

I don’t hate JavaScript because of JavaScript Framework Bingo.

I don’t hate JavaScript because because I think significant whitespace is a perfectly reasonable way to block code.

I hate JavaScript because the platform once forced me into Java’s devils’ bargain that included rewriting everything I had already written.

Have a thirty year old, battle tested cartographic projection library?

Rewrite that sucker in JavaScript so you can reproject 4326 into Web Mercator and UTM.

Half-ass the other methods, because who really needs support for any other coordinate systems anyway?

Partially implement the geometry algebra in JavaScript you need to split some polygons or compute their union?

Ditch on the crazy fractal of numerical edge cases that start popping out when you implement the full 9IM.

Need to translate data from esoteric geospatial format elevation format to another?

Like plastics, my boy, reimplementing format translators for every computing platform is still a growth industry.

Want to handroll a JavaScript implementation of a lidar compression and decompression engine?

Good work if you can find someone who’ll pay for it.

You get my point, but we often keep our head down and climb the JavaScript reimplementation hill in front of us.

Do I really look like I climb hills?

All of the platforms I described earlier aren’t equal, but they all have something that can target them

LLVM

LLVM is just a name now, although low-level virtual machine might once have been a fitting acronym.

It has quietly infiltrated nearly all of your developer tools in recent years.

Native mobile developer tooling and continual investment in LLVM by Apple pushed it over the threshold of nearly everyone building off it directly or supporting it.

LLVM is compiler technology. It’s different than others in that it breaks the compilation problem up into pieces.

The typical compiler scenario we all might think of is having a C++ frontend and a machine backend that is x86 assembly.

Or how about a Python frontend and a JVM backend?

Or, even better for me, a C++ frontend and a JavaScript backend!

This exists today.

You can use the LLVM toolchain to compile C/C++ into LLVM intermediate representation and target JavaScript as a backend.

The tools to google are asm.js, WebAssembly, and Emscripten.

These tools are in use today.

My company, Hobu, Inc used them to compile the C++ LASzip lidar compression library into a javascript module that only performs a few multiples slower in a browser than it does on bare metal.

Every release tick of the browser and every release tick of the compiler stack has made that multiple decrease. We haven’t had to do any work. Just recompile.

It’s not a panacea. Compiling the library to JavaScript actually forced us to completely refactor the LASzip library in a way that was probably equivalent to porting it to a new platform. At the time, Emscripten implemented every c++ virtual table dereference as a function call in JavaScript.

This meant a simple dereference of a variable was three or four function calls.

It took a month to refactor it, and along the way, we added open source niceties like tests.

If we had simply ported it to JavaScript, we’d have burned a month forking the library with yet another implementation.

More chances for bugs.

More software to silently fall out of date.

More compatibility to lose.

LLVM in JavaScript – and all its other backends – enables what I call the “Rotten Core” development model .

Programmers are very good at detecting rotten code.

It’s quite easy, really.

Anything not written by they themselves is rotten.

More seriously, Rotten Core plus the LLVM stuff means we don’t have to make the platform devils’ bargain.

We can abide with the embedded lessons that decades’ old software libraries can provide our modern software without abiding their outdated APIs.

https://twitter.com/arnicas/status/856569143021236224

Lynn doesn’t like GDAL’s API, but usable here means usable to her in her context.

What Lynn really wants an abstraction layer to GDAL’s power within the context of language and programming idioms she knows.

rasterio is the most idiomatic Python raster processing library. It’s all GDAL underneath, but it protects users from the most jagged bits by hiding the necessarily complex GDAL API.

rasterio is Rotton Core in the sense that it hides all the rotton-to-me stuff and wraps it with a convenient and idiomatic API that has meaning within my own programming context.

With LLVM, Emscripten, and JavaScript, we can start doing the same thing in our favorite browser platforms too.

I took over maintenance of the Proj.4 library after Frank Warmerdam moved on.

Proj.4 is a little unique in our industry because it is so old – almost 35 years.

It’s everywhere, and it’s going to continue to be everywhere for an unpredictably long time.

The reason why is it solves such a fundamental problem of our domain that once solved, doesn’t really need to be solved over and over again.

Software libraries that condense knowledge of data formats and math – think GDAL and Proj.4 – are going to be around for a very long time. Longer than any of our careers.

They are heirlooms that will be patched up, recompiled and retargeted, and gussied up for their new usages.

It is folly to reimplement them just to reach a new platform, especially now that Rotton Core programming can work in JavaScript in the same way it might in any native language.

JavaScript is just another platform, and the tools are maturing enough that we can treat it like any other one we’ve been building boring geospatial software on for decades.

I hope you’ll think of LLVM, Emscripten, and Rotten Core before you dive into some brownfield reimplementation of an open source library that’s been around for 20 years. Don’t think all those pockets aren’t on there for a reason.